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Introduction 

In spring 2014, more than one million students in public schools across the coun- 
try participated in field tests of new assessments developed by the Partnership 
for Assessment of Readiness for College and Careers (PARCC). 1 These assessments 
are designed to measure students’ knowledge in English language arts (ELA) and 
mathematics as defined by the Common Core State Standards. The field tests were 
conducted to ensure the assessments— including new computer-based assess- 
ments— are valid and reliable, while providing an opportunity for state and local 
administrators to gain insight into the management and use of new technologies to 
support computer-based testing. 

Massachusetts is one of the 10 states currently in the PARCC consortia. Roughly 
81,000 students in districts and schools across the Commonwealth participated in 
the spring 2014 field tests. The field test was an initial step in the two-year time- 
line established by the Board of Elementary and Secondary Education (BESE) 2 
to consider whether PARCC will replace the existing statewide Massachusetts 
Comprehensive Assessment System (MCAS). Prior to the 2014-15 school year, all 
Massachusetts districts were given the option to administer either PARCC or MCAS 
to students in grades 3-8. Districts choosing PARCC could use either a computer- 
based or a paper-based test, and elect to do so on a school-by-school basis. In 
fall 2015, BESE plans to vote on whether to adopt computer-based PARCC assess- 
ments statewide. If BESE votes to adopt PARCC, the first required administration of 
PARCC for students in grades 3-8 would occur in spring 2016. In future years, there 
is potential for PARCC to serve as the high school diploma competency assessment 
in grade 10. 3 


PARCC Trials in Burlington and Revere 

During the spring 2014 field test, a certain number of classrooms in a given school/ 
district were selected by PARCC to take either a paper-and-pencil or computer- 
based version of the PARCC assessment. Of the 81,000 students who took a PARCC 
field test in spring 2014, about 70 percent of these students participated in a com- 
puter-based administration. Two districts— Burlington Public Schools and Revere 
Public Schools— volunteered to administer PARCC ELA and math computer-based 
tests to all students in testing grades, or in selected schools. 

In Burlington, all students enrolled in grades 3-8 and 10 participated in PARCC field 
tests; in Revere, two elementary and one middle school of the district’s 11 schools 
administered PARCC tests school-wide (see Table 1). All tested students took both 
the Performance-based Assessment (PBA) and the End of Year (EOY) 4 assessments. 
The decision of these two districts to administer PARCC tests district- or school- 
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1 For more information visit: http://www.parcconline.org/about-parcc. 

2 Massachusetts is the only PARCC consortia member implementing this two-year timeline. All other 
consortia states will be implementing PARCC tests in spring 2015. 

3 Massachusetts Department of Elementary and Secondary Education. (May 2014). MCAS or PARCC. 
Retrieved from: http://www.doe.mass.edu/mcas/2014/news/MCASorPARCC.pdf. 

4 The PBA is a summative test that occurs after 75 percent of the school year to measure student 
knowledge. The EOY occurs after the completion of about 90 percent of the school year. Results 
from both tests are combined to produce students’ summative assessment scores. 
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| Burlington 

Revere 

Total schools tested 

6 schools 

3 schools 

Total students tested 

2,200 students 

950 students 

Total enrollment 

3,579 students 

6,831 students 


wide presented an opportunity to examine PARCC admin- Table 1. Student enrollment and participation in Burlington 

istration at a scale reflective of actual testing procedures, anc * Revere 

providing critical information on administration practices 
and pitfalls prior to key decision-making points at state and 
local levels. 

The emergence of Burlington and Revere as leaders in a 
process designed to examine the adaptability and use of 
computer-based technologies in schools is not a surprise. 

Both districts have devoted resources over the last several years to improving technology integration in classroom instruction. 
In partnership with their local municipalities, the districts have developed sophisticated information technology (IT) systems 
that include fiber optic network connections between school facilities, high-speed connectivity, and the latest generation 
devices to support student learning. 

In addition, the importance of both districts’ commitment to recruiting and hiring highly-qualified IT staff cannot be over- 
stated. Recognizing staff capacity to support technological innovation as a clear priority, the districts have been creative in 
finding funds to support new positions. Burlington Public Schools shares with the Town of Burlington a director-level IT posi- 
tion that oversees technology infrastructure. In Revere, the district has applied for the maximum amount of services or funding 
available through the Universal Service Program for Schools and Libraries (E-rate) for several years. 5 The district uses these 
funds for annual upgrades or expansion of equipment and services, freeing up district funds to support staff positions. IT staff 
in both districts have a diverse set of roles, including building and maintaining central systems, facilitating training on new 
equipment and devices, and working collaboratively with classroom educators on technology integration issues. 

However, despite a shared emphasis on IT investments, substantial differences exist between Burlington and Revere in the 
educational challenges they confront. Burlington Public Schools is a suburban district with a less diverse student population 
compared to the Commonwealth’s general student population. The district is high-performing with approximately 94 percent 
of high school students completing their high school diplo- 


ma in four years. 6 Revere Public Schools in a large urban 
district, and enrolls a more diverse student population 
than that of the state. Approximately 82 percent of stu- 
dents are classified as high-needs, nearly double the state 
average. 7 The district has a four-year graduation rate of 
approximately 77 percent. 8 Table 2 provides demographic 
data for each of the districts, as well as for Massachusetts. 


Table 2. District populations, as compared to state totals 



Burlington 

Revere 

Mass. 

Enrollment 

3,579 

6,831 

955,739 

Fligh-needs students 

25% 

82% 

49% 

Four-year graduation rate 

94% 

77% 

85% 


Examining strategies employed by Burlington and Revere can provide a fuller sense of the range of options available to, 
and challenges faced by, other districts considering the use of computer-based tests. For example, districts may consider 
new investments needed to improve IT infrastructure, as well as additional staff capacity and expertise. It was this learning 
opportunity, in fact, that motivated both districts to participate in district-and school-wide trials. As Revere Public Schools 
Superintendent Paul Dakin noted: “I wanted to tackle some [school] management questions. For example, should a principal 
or an assistant principal do scheduling? And once we figure this out, I want principals who did this to tell the other principals 
[in my district] about it.” Eric Conti, Burlington Public Schools’ Superintendent, expressed value beyond his own district’s prep- 
aration process: “I saw the value for us and others. . .1 wanted to be device-agnostic. I wanted to involve students in the pro- 
cess as well; they’re great consumers.” Documenting and sharing these types of lessons can raise important questions about 
the effect of these tests on school operations, particularly teaching and learning. 


5 The nation’s largest education technology program administered through the Federal Communications Commission, the E-rate Service Program provides 
discounted telecommunications, internet access/service and internal connections to schools and school districts— including broadband service. 

6 Massachusetts Department of Elementary and Secondary Education. (2013). School and District Profiles: 2013 4-year district graduation rate report for all 
students. Retrieved from: http://profiles.doe.mass.edu/state_report/gradrates.aspx 

7 Massachusetts Department of Elementary and Secondary Education. (2013). School and District Profiles. Retrieved from: http://profiles.doe.mass.edu/ 
profiles/student.aspx?orgcode=02480000&orgtypecode=5&leftNavld=305& 

8 Massachusetts Department of Elementary and Secondary Education. (2013). School and District Profiles: 2013 4-year district graduation rate report for all 
students. Retrieved from: http://profiles.doe.mass.edu/state_report/gradrates.aspx. 
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This case study, prepared by the Rennie Center for Education Research & Policy, examines the experiences of Burlington 
and Revere Public Schools in administering PARCC field tests district- or school-wide. It offers insights for other districts 
considering the implementation of computer-based assessments in technology use and adaptability, decision-making, 
scheduling, staffing, and student data management. The case study activities were guided by a steering committee includ- 
ing the superintendents of the Burlington and Revere Public Schools, the Burlington Educators’ Association and the Revere 
Teachers Association, along with the Massachusetts Department of Elementary and Secondary Education, the Massachusetts 
Association for School Superintendents, and the Massachusetts Teachers Association. 


Research overview 

The potential use of computer-based assessments has raised concerns from educators, policymakers, and parents about informa- 
tion technology infrastructure in school districts and the preparation of staff and students to use new technologies for assess- 
ment purposes, and the potential impact of testing activities on core school functions, particularly teaching and learning. This 
case study documents lessons learned in the district- or school-wide administration of computer-based student assessments in 
the Burlington and Revere School Districts during the spring 2014 PARCC field test. The case study focused on three research 
questions: 

■ What was the preparation process for district- and school-wide implementation of an online student assessment, including 
decision making on scheduling, staffing, professional development, technology and hardware, and test materials manage- 
ment? 

■ What successes and challenges were discovered through administration of computer-based assessments? What role did test- 
taking instructions, appropriateness of physical space, facility with technology, as well as the quality of professional develop- 
ment play in these discoveries? 

■ What was the reported experience of student test- takers? What were educators’ (e.g., school leaders, teachers, test administra- 
tors) perspectives on students’ test-taking experience? 

To adequately answer these research questions, the Rennie Center team conducted the following research activities in both 
Burlington Public Schools and Revere Public Schools: 

■ Interviews with district leadership and district technology staff; 

■ Focus groups with principals, teachers and students from all testing schools (i.e., elementary, middle and high schools), 
accounting for nearly 170 leaders, teachers and students; 

■ Survey of staff prior to the administration of the PBA tests (i.e., pre-test survey); and 

■ Survey of staff after the administration of the PBA tests (i.e., post- test survey). 

Almost 400 staff members, across the two districts, responded to the pre-test survey and about 250 to the post- test survey. The 
overall response rate for the pre-test survey was about 60%; 99% of Revere educators in the testing schools completed a survey. 
In Burlington, where all educators across the district received the questionnaire, 55% of educators completed a pre-test survey. 
For the post-test survey, response rates were about 42% overall (88% in testing schools in Revere; and 34% in Burlington). At 
Burlington High School, fewer teachers were directly involved in test administration since only 10th grade students were tested 
in English language arts and math, but not in other subjects. Additional information about the survey respondents is provided 
in Appendix A. 

All data was then compiled and analyzed according to key implementation issues. The findings section presents themes that 
were common across respondents regardless of role or district affiliation, and offers more detailed descriptions of specific imple- 
mentation issues. 


Rennie Center for education research & policy 
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Case study findings 

The research findings below describe the preparation for and administration of PARCC field tests— both Performance-based 
Assessment (PBA) and End-of-Year (EOY) administrations— in Burlington and Revere. Given the nature of findings discussed, it 
is important to note that both Burlington and Revere possess sophisticated information technology (IT) systems, with features 
and resources designed to support student learning. In documenting and analyzing the strategies employed by these districts, 
the Rennie Center team has chosen to highlight challenges that will likely be common to other districts that elect to imple- 
ment computer-based PARCC at scale, including: 

■ technology infrastructure, 

■ device use, 

■ scheduling, 

■ staffing, 

■ training educators, and 

■ student testing experiences. 

Within each issue area, findings are organized to provide insights on Burlington’s and Revere’s planning processes; test admin- 
istration; educators’ experiences and perspectives; and these districts’ decisions for spring 2015 PARCC testing. 


Technology infrastructure 

Burlington and Revere have both been working for many years to maintain and upgrade their IT systems to more fully inte- 
grate technology into their classrooms. Among the most important features of their existing infrastructure, both districts have 
fiber optic connections between school buildings that provide high-speed internet service and a connection to central servers. 
This type of fiber optic connectivity accommodates bandwidth in excess of the minimum requirements specified by PARCC, 
and allowed both districts to easily access downloaded test materials at each testing site. As a result, both districts were inter- 
ested in going beyond the request for a limited number of classrooms and a paper-and-pencil administration to participate 
in the field test using computer-based technology, and offered to do so district- or school-wide. In addition, they utilized dif- 
ferent testing procedures to learn about practical implementation issues. Table 3 below provides an overview of technology 
infrastructure features in these two districts. 

■ How did these districts plan for technology use during the spring 2014 PARCC trial? When opting into the district- and 
school-wide trials, Burlington and Revere agreed to vary their testing procedures and device use to learn as much as pos- 
sible about practical implementation issues. Revere conducted the trial by proctor caching. 9 Burlington opted for live 
stream test administration. 

The use of central servers linked to testing sites through high-speed connections allowed both districts to update devices 
through a centrally-administered program. This allowed district-level IT staff in Burlington to centrally access device 
applications from all buildings to update mobile devices in batches (about 30 devices at a time). Similarly, because of 
sophisticated infrastructure, district IT staff in Revere had an analogous centrally-administered process to update laptops 
and desktops, with much of the preparation managed by district IT staff and executed by school-level IT staff. 

■ What happened during the trial? Both districts reported relatively few concerns in successfully being able to download 
materials; the bandwidth used for test administration did not compromise internet use for other educational purposes in 
schools. However, almost all test sessions were affected by test materials freezing or spooling for an extended period of 
time. These issues likely resulted from problems with how test materials, developed by the PARCC consortium interacted 
on Pearson technology applications, created as part of a suite of test administration products for PARCC tests (see addi- 
tional discussion in Devise Use section below). Many Pearson-developed resources were released late, raising questions 
about whether products had been sufficiently tested for the scale at which they were being used, and forcing many dis- 
trict IT staff in both districts to scramble to complete needed updates to devices. District IT staff and test administrators 
were able to resolve most of these problems with a reboot of test materials. 


9 Proctor cache is a test administration approach where all test materials are downloaded to a local server, from which students access test materials dur- 
ing a test session. Live streaming is a test administration approach where all test materials are streamed live from the internet during the test session. 
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Table 3. Technology infrastructure resources 



Burlington 

Revere 

Hardware configuration 

Central server with wired connectivity to each 
building 

Central server with wired connectivity to each 
building 

Network configuration 

Fiber optic network exists between buildings 
(includes all schools and town facilities) 

Fiber optic network exists between buildings 
(includes all schools and town facilities) 

Test administration approach 

Live stream 

Proctor cache 

Internet connectivity: Bandwidth capacity 

500 MB 

500 MB; increasing to 1 GB for 2014-15 school year 

Bandwidth used during test sessions 

25-30% of total bandwidth 

25-30% of total bandwidth 


More significant challenges arose for Burlington due to the decision to live stream test materials; student test data was not 
being received due to a programming error affecting communication between Chromebooks and the Pearson server. Neither 
district had significant concerns after the first few days of testing, which speaks to the relative strength of their infrastructure 
plans. However, these issues must be resolved for the next administration of PARCC. 

■ What were educator perspectives on technology 
implementation? Overall, educators expressed 
confidence in their schools’ technology resources, 
according to survey data. Figure 1 summarizes 
educator perspectives on school technology 
resources collected pre- and post-PBA adminis- 
tration (see Appendix A for more data). Post-test 
survey findings reveal increased percentages of 
educators in Burlington reporting technology was 
not ready after PBA test administration; these 
results may reflect frustration with initial techno- 
logical glitches. 


■ What decisions about infrastructure are the dis- 
tricts making to prepare for 2014-15 implementa- 
tion? Given the sophisticated technology infra- 
structure these two districts possess, Burlington 
and Revere are not planning significant changes 
to their IT systems in light of information gained 
from the district- and school-wide trials. Revere 
decided in spring 2014 to increase its available 
bandwidth to 1 GB, and to purchase additional wireless access points to support improved wireless coverage in all school 
buildings. Both districts have ruled out live streaming test materials; both will use proctor caching to administer test 
materials. 


See text box below on Technology Infrastructure for a summary of district decisions, and questions for other districts to con- 
sider concerning IT infrastructure issues. 


TECHNOLOGY INFRASTRUCTURE 

District decisions for 2014-15 

Key questions for other districts 

Burlington and Revere will opt for: 

■ Computer-based testing; and 

■ Proctor caching all test materials. 

■ How is network access set up in your district? 

■ Where are district/school servers located? 

■ Are buildings networked to each other? A central server? Does each 
building have its own server? 

■ What is the amount of bandwidth available? To the district? To each 
building? 


Figure 1. Percent of educators reporting on school’s technology 
readiness.* 



I Strongly Agree 
Agree 
Disagree 

I Strongly Disagree 


Pre-Test Post-Test 
BURLINGTON 


Pre-Test Post-Test 
REVERE 


* Figure 1 reports on the percent of educators that reported at least some 
knowledge on technology readiness, and does not include “I don’t know ” 
responses. 
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Device use 

Burlington and Revere used different devices as part of the spring 2014 PARCC 
trial. Revere conducted the trial largely on PC desktops and laptops; Burlington 
purposefully chose to use a variety of devices (i.e., laptops, desktops, iPads, 
Chromebooks) to learn about their functionality with test content, and with 
students of different ages and abilities. This variation between the districts’ test 
administration plans produced some of the most illustrative examples of the 
challenges in implementing PARCC assessments district- and school-wide. 

■ How did districts plan for device use during the spring 2014 PARCC trial? 
Burlington’s IT staff developed a testing schedule to facilitate the use of 
multiple devices. For elementary schools, all students in each grade used 
the same device as follows: 

■ Grade 3: Windows/PC desktop 

■ Grade 4: iPad 

■ Grade 5 and 6: Chromebook 

All three devices were used in grades 7-8 and 10. Revere used Windows/PC 
desktops and laptops in all testing grades. 

Burlington’s Director of Information Technology led a team of five other IT 
professionals to prepare all devices for testing. This included installing a 
mobile device management (MDM) application on both Chromebooks and 
iPads. The MDM application is used to ensure mobile devices— principally 
Chromebooks and iPads— would run programs needed to display test 
materials (e.g., TestNav) and students would not be able to navigate away 
from test materials. 

■ What happened during the trial? While Burlington found that it had a suffi- 
cient number of devices for their district-wide trial, Revere used computers 
from another building to complete the trial in selected schools. The need to 
use the district’s existing inventory will have implications for implementing 
computer-based testing district-wide in Revere in spring 2015. See Table 4 
for a detailed comparison of device use in each district, device count, and 
staff hours dedicated to preparing devices for test administration. 


Common planning 
processes 

Inventoried equipment. District- 
level IT staff inventoried devices, 
and determined if they met PARCC 
specifications. 

Configured devices. IT staff deter- 
mined needed upgrades, and devel- 
oped appropriate workplans. 

Designed test schedule. District and 
school leaders created a test adminis- 
tration schedule, factoring in device 
count and location. 

Prepped devices for tests. IT staff 
installed software for test administra- 
tion on all devices. 

■ In Revere, IT staff programmed 
laptops and desktops in batches, 
downloading files for test admin- 
istration. 

■ In Burlington, IT staff invested 
significant time installing MDM 
programs on mobile devices to 
run test materials in single app 
mode (e.g., TestNav) 

Tested equipment and connectiv- 
ity. With updates/installations com- 
plete, IT staff tested all devices to 
determine if internet service would 
be interrupted during test sessions. 


Table 4. Device supply and use in PARCC trials 



Burlington 

Revere 

Types of devices used for test administration 

■iPads 

■ Chromebooks 

■ Desktops (PCs and Macs) 

■ Laptops (PCs) 

■ Desktops (PCs) 

Time invested by technology staff to prepare 
devices 

100% of Director for Technology Integration 
for three weeks prior to trial, with additional 

500 staff hours across five staff members on 

district tech team 

100% of four FTEs for a period of three weeks 

District-wide device count 

Approximately 4200 (800 PCs and laptops; ap- 
proximately 3400 iPads/Chromebooks) 

Approximately 6000 (3500 PCs and laptops; 
approximately 2500 iPads) 

Proportion of devices used for test 
administration 

100% of computer labs in testing schools; 

100% of Chromebooks, and 25% of iPads 

100% of devices across testing schools plus 
two computer labs’ worth of devices from 
other schools 

District rotation to replace equipment 

Three-year rotation on all devices 

3-4 year old laptops in most classrooms and 
labs, some devices 6 to 7 years old 


6 




■ What were educator perspectives on device use? Based on survey 
data, almost all educators in these two districts who had a role 
in test administration reported that devices were affected by 
various technology challenges during test sessions (see Figure 2; 
more data available in Appendix A). A high percentage of educa- 
tors serving as test administrators reported a student’s device 
not working during a test session or a device losing connectiv- 
ity; these reported percentages were higher in Revere. Given the 
relative ease with which these districts handled batch updating 
of devices and downloads of testing materials, and the extensive 
quality controls that district-level IT staff performed (e.g., testing 
TestNav, confirming appropriate versions of software applica- 
tions were downloaded), the challenges with device use would 
suggest that Pearson computer-based testing platforms and the 
test materials that ran on them were not ready for this intended 
scale of use. 

■ What decisions are Burlington and Revere making about devices 
to prepare for spring 2015 testing? District IT staff in Burlington 
and Revere agreed that PC desktops performed best in terms of 
loading and running test materials. Desktop computers required 
the least time investment for preparation, given that most software could be loaded with batch updates and produced 
the fewest technical problems and other interruptions during test sessions. Chromebooks and iPads experienced more 
interruptions during testing, and test materials were more difficult to load. 

However, neither district is seriously considering purchasing additional desktop computers solely to support test admin- 
istration. All planned technology purchases will be geared toward enhancing classroom instruction rather than solely 
addressing test administration needs. In spring 2015, Burlington is likely to use mobile devices, since the district’s inven- 
tory is largely comprised of iPads; Revere is considering the purchase of Chromebooks to ensure a sufficient number of 
devices. 


See text box below on Device Use for a summary of these district decisions, and questions for other districts to consider 
regarding device use. 


DEVICE USE 

District decisions for 2014-15 

Key questions for other districts 

Burlington will opt for: 

■ Using the entire inventory of mobile devices to support computer- 
based testing and limit computer-lab based sessions. 

Revere is considering: 

■ Purchasing additional Chromebooks to ensure the district has the 
number of devices needed to administer PARCC at scale. 

■ How many existing devices are available for test administration? 

■ What type of devices do these include? 

■ Where are these devices located? And for what are they typically 
used? 

■ How many devices are required for instruction that will be occurring 
during test administration? 

■Are software and hardware features on these devices compatible with 
PARCC test specifications? 


Figure 2. Percent of test administrators reporting 
technology challenges in test sessions* 


100 



i ^ i ^ i ■ 

BURLINGTON REVERE 


■ Student devices did not appear to be working. 

■ Student devices lost internet connectivity. 

* Figure 2 reports on the percent of educators with o test 
administrator role during the PBA reporting on the 
functionality of students devices during test sessions. 
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Scheduling 

MCAS (or any paper-based test) administration is largely accomplished across a few days with most students in a grade, or 
school, taking the test at the same time. Devising a testing schedule for a computer-based test is almost entirely dependent on 
the number of existing devices that can be used to test students at any one given time. A major challenge in both districts was 
creating a school-level test administration schedule that maximized the number of available devices, yet minimized school 
disruptions. 

■ How did districts and schools plan and schedule test sessions? I n 
Revere, planning for PARCC tests was a collaborative endeavor 
between lead district IT staff, school-level IT staff, and school princi- 
pals. Principals and the Assistant Superintendent set the test admin- 
istration schedule in one all-day planning session held over winter 
break in 2013-14. This allowed each principal sufficient time to work 
with district- and school-level IT staff to ensure an adequate device 
inventory on-site to accommodate the testing schedule. With this 
information about testing schedule and device needs, district IT staff 
created workplans for updating devices to meet PARCC specifica- 
tions. 

District and school staff in Burlington pursued a very similar multi- 
step collaborative planning process as Revere for the PBA, but it 
was led and managed by the district-level IT team, with feedback 
on scheduling from principals. For the EOY, however, Burlington 
principals managed the school schedule, for which they expressed a 
preference. As one principal noted: “the more control you can have 
on schedule and time allocations the better. You can make school- 
level decisions”. 

■ What happened during the trial? To test the approximately 2,200 
students who participated in the field test, Burlington used nearly 
all 15 days of the PBA test administration window. In testing far 
fewer students, Revere used about the same number of testing days 
given the more limited number of devices available across the three 
testing schools. In short, Burlington’s larger inventory of devices 
enabled a greater number of students to be tested in a shorter num- 
ber of testing days. 

Successfully scheduling PARCC administration required these dis- 
tricts to determine which school resources and educators’ time 
could be made available to support test administration and still 
allow for other core school functions to continue. Ensuring adequate 
staffing for regular educational activities was, perhaps, more complex than staffing test administration. Space was also a 
concern. For example, computer labs in testing schools in both districts were taken off-line for parts of many days during 
the PBA and EOY testing windows. 

Scheduling test sessions where students’ course-taking patterns vary considerably was even more complex. Middle and 
high school teachers reported experiencing class sessions where students were missing due to the testing schedule, or 
having class sessions run on different timing. In focus groups, some teachers in non-tested subjects reported significant 
disruptions to instruction, and reported these disruptions as obstacles to adequate coverage of course curriculum during 
the test administration period. Overall, district and school leaders acknowledged that the desired goal of “business as 
usual” during the three-week testing period was difficult to achieve. 

■ What were educator perspectives on scheduling? Similar to data on school-level technology readiness, educators in 
Burlington reported less favorably on their school’s administration/logistics plan after PBA administration than before 
(see Figure 3; additional data available in Appendix A). It is important to note that Burlington experimented with a number 


Planning for PARCC test 
accommodations 

Educators in these two districts encoun- 
tered several challenges when attempting 
to plan for students who routinely test with 
accommodations. First, the PARCC field 
tests only made a limited number of test 
accommodations available. For example, 
during the PBA, only “read aloud” and 
“scribe” accommodations were available 
as computer-based adaptations to test 
materials. 

Next, documentation from Pearson was not 
explicit on how computer-based accom- 
modation features would work with test 
materials, making educators unsure of 
whether students should receive the same 
accommodations as a prior year’s MCAS 
administration. 

Finally, once student testing decisions were 
made, a modified test administration sched- 
ule needed to be crafted. Staff in both dis- 
tricts followed MCAS test administration 
protocols in which students with accommo- 
dations are often tested in a different room/ 
time, including students with read-aloud 
and scribe accommodations, creating addi- 
tional demands for space, devices and staff. 
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of scheduling strategies to determine the best approach moving forward. In Revere, the opposite scenario held true; more 
educators reported favorably on school administration/logistics plans after PBA administration was completed. 

More than half of all educators reported that school resources like instructional space, equipment, and facilities were 
substantially impacted due to the PARCC testing schedule (see Figure 3 below, and additional data in Appendix A). For 
example, in Revere, where computer labs were more frequently used for test sessions, a higher proportion of educators — 
nearly three-quarters— reported that school resources were substantially impacted. 


Figure 3. Percent of teachers reporting significant impact on school-level logistics due to PARCC testing.* 



ELEMENTARY MIDDLE HIGH 


(Burlington only) 


■ Rooms, equipment, or facilities availability 
was impacted by the PARCC PBA. 

■ Class time/schedule was impacted by the 
PARCC PBA. 


*Figure 3 reports on the percent of educators reporting some knowledge on school administration and logistics readiness , and does not include “I don’t 
know” responses. 


A high proportion of educators across both districts reported that class time or their class schedule was substantially impacted 
by PARCC testing. About 70 percent of educators, across all schools, reported this to be the case (see Figure 4 below and 
Appendix A for additional data). 10 In focus groups, teachers discussed that quiet classroom space— located away from the caf- 
eteria or a playground— with sufficient connectivity for multiple devices was also at a premium during the test administration 
window. 


Figure 4. Percent of educators reporting substantial impact on school-level logistics due to PARCC testing.* 


100 

80 

60 

40 

20 

0 



Pre-Test Post-Test Pre-Test Post-Test 


BURLINGTON REVERE 


■ Strongly Agree 

■ Agree 
Disagree 

■ Strongly Disagree 


*Figure 4 reports on the percent of educators reporting significant impact on school-level logistics. 


10 To note, the sample of Burlington teachers in Figure 3 includes Burlington High School teachers. Sophomores at the high school participated in this 
PARCC trial, and also took the 10th grade MCAS exam during two consecutive three-week periods in March. As such, these students’ schedules varied the 
most of any of the schools in the case study. The Burlington teacher data on school-level logistics may be linked to teacher perceptions’ of these more 
intensive test administration schedules used at Burlington High School. 
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■ What decisions are Burlington and Revere making in regard to school-level test administration? Both Burlington and 
Revere are focused on scheduling test sessions to minimize the number of days across which test administration stretches 
in spring 2015. Revere is weighing several scheduling scenarios, including scheduling two testing sessions per student per 
day (e.g., a student would take both a math and ELA session in a given day). In Burlington, the district will predominantly 
rely on their inventory of mobile devices plus laptops/desktops, totaling about 4000+ devices, to test students in grades 
3-8 in as few days as possible. Utilizing more devices from district inventories, both districts are likely to test far more than 
the 600 students (Burlington) and 175 students (Revere), respectively, that were tested at one time during the spring 2014 
trial. 

See text box on Scheduling below fora summary of these district decisions, and questions on planning the school schedule 
during test administration for other districts to consider. 


SCHEDULING 

District decisions for 2014-15 

Key questions for other districts 

Burlington and Revere will attempt to complete PARCC testing in 
fewer days. 

Burlington will opt for testing all students in a given grade in one day, 
maximizing mobile devices in use. 

Revere will likely schedule two sessions per day so that students are 
taking both a math and ELA session in one day. 

■ How many existing devices are available for test administration? 

■ What type of devices do these include? 

■ Where are these devices located? And for what are they typically 
used? 

■ How many devices are required for instruction that will be occurring 
during test administration? 

■ Are software and hardware features on these devices compatible with 
PARCC test specifications? 


Staffing 

Administration of computer-based tests requires different staffing arrangements than what is needed for paper-and-pencil 
test administration. Both paper-and-pencil tests and computer-based tests require staff to be responsible for scheduling test 
sessions, assigning students, coordinating the staffing of test sessions, ensuring that security protocols for test materials are 
followed, and conducting the test (i.e., staff read instructions and monitor student progress). 

With computer-based tests, however, scheduling a test session in the test administration platform and assigning students to 
each session are completed online. These tasks may require staff with different skills and knowledge, including familiarity with 
a student information management system. Staff supervising test sessions need to be able to “initiate the test session” by 
making sure all students can log on and access the test materials, as well as deal with basic technology issues that may occur. 
In addition, staff with technical expertise are needed to prepare devices for test sessions and troubleshoot more complicated 
technology issues. Staffing decisions ultimately involve figuring out what knowledge and experience is required for each task, 
which and how many staff should be assigned to perform each task. 

■ What staffing plans did Burlington and Revere use? Burlington and Revere used somewhat different staffing models for 
planning for the PBAtest administration (see Table 5). In Burlington, the district’s Director of Technology Integration did 
the majority of scheduling; the Director of Student Information Management created test sessions, and assigned students 
to these sessions in Pearson ACCESS, the online database management system used for managing student test data. 
Further, Burlington’s Director of Information Technology led a team of five other IT professionals to prepare all devices, 
district-wide, for testing. This included installing a mobile device management (MDM) application on all mobile devices 
(described in detail in the Device Use section above). In Revere, planning for PARCC test administration was a collabora- 
tive endeavor between lead district staff and school principals. School level staff worked with district IT staff to plan how 
to use available devices, create the testing and training schedule and coordinate the test administration. 
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Table 5. Staffing 


Burlington 

Revere 

Test administration: Staff in the 

classroom 

District technology staff initiated test sessions; 
school staff supervised and monitored students 
and provided basic troubleshooting. At Burlington 
High School, permanent and per diem substitutes 
(often retired teachers), who typically serve as 

MCAS proctors, performed these tasks. 

Teachers served as test administrators, and were 
responsible for initiating test sessions. 

Test administration: Staff at the 

school 

At least two technology staff on-site at each build- 
ing during test sessions (one district-level tech lead 
and a school-level library and media specialist). 

Two or three technology staff leads on site at each 
building during test sessions (a mix of school-level 
tech leads and at least one district-level tech lead). 

Number of technology staff district- 
wide 

Eighteen staff: district team of 6 (including district/ 
town IT positions), plus two school-level positions 
per building and additional district-level director. 

Nine staff: combination of district-level and school- 

level staff. 


■ How did Burlington and Revere staff test sessions? In both districts, staffing decisions focused on the number of staff 
needed to administer the tests properly, to handle technology issues, and what expertise these staff needed. During the 
first days of the PBA administration, each district assigned two school staff to each testing session. One person supervised 
and monitored the test session, reading the instructions to students and ensuring test protocols were followed. Another 
staff member was typically assigned specifically to be available for “first-line” technology support. However, in many 
cases, this second staff member was assigned to as many as two to three classrooms. This support ranged from assisting 
students with log in problems, swapping equipment as needed, rebooting devices, and dealing with freezing or spooling 
issues. In addition, each district assigned two to three technology staff (both district- and school-level IT staff) to a testing 
building to be available for more complicated technology issues. 

Although staffing levels were similar, Burlington and Revere handled test session initiation in different ways. In Burlington, 
school- and district-level technology staff initiated the test session in the Pearson software interface to allow students to 
log-on. After starting the online test session, classroom teachers, special education teachers, and guidance counselors 
supervised test sessions , 11 while library and media specialists, district-level IT staff, and instructional technology special- 
ists provided basic technology support. 

In Revere, classroom teachers had responsibility for initiating a test session, and prompting/assisting students to login, 
and monitoring test protocols. 

Student information management in PARCC testing 

In implementing district- and school-wide trials, Burlington and Revere had to create test sessions in Pearson ACCESS — the 
online student information/database management system for PARCC tests — for all testing students. A key decision was how 
to best manage this process. While ESE prepared formatted files for districts to use in uploading student data, districts then 
needed to create the actual test sessions from grade-level files and make adjustments for any students requiring accommodations 
or make-ups. As a district technology staff member reported: “touching each student [record] was incredibly time-consuming, 
and we wanted to make sure we got it right.” 

In Burlington, student assignment to test sessions was managed centrally by the district’s Director of Student Information 
Management; principals had a role in the quality control of these assignments. This process took the equivalent of three weeks 
of staff time. In Revere, principals completed these processes, resulting in about 20-30 hours of principals’ time each week for 
about three weeks. District leaders reported that the entire process was quite time-consuming and required more time than was 
initially expected. 

In both districts, staff responsible for creating test sessions did not receive extensive training on the Pearson ACCESS system; 
Pearson documentation was described as “thin.” Principals in Revere responsible for creating test sessions did not have much, 
if any, previous experience in online data management. With computer-based testing, these student information management 
responsibilities will be more central to test administration. If computer-based testing is adopted, training in online data man- 
agement will be essential for district and school leadership. 
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■ What happened during the trial? Staff in both districts almost unanimously reported that technology staff and teach- 
ers seamlessly worked together to get students re-logged into test sessions in the case of technology delays or failures. 
Districts began from the first days of testing to routinize some of the processes that worked well to support teachers in 
new roles. For example, principals in Revere reported that most classroom teachers who served as test administrators 
became adept at serving as first-line technology support, knowing when to re-start test sessions due to continual spooling 
or freezing. Teachers who conducted test sessions on the first day of testing then became responsible for troubleshooting 
for a test session occurring on a subsequent day of testing. 

■ What were educator perspectives on their role in staffing test sessions? Principals and teachers alike reported having “no 
expectations of what computer-based testing would be like until that first session.” Many teachers approached their test 
administration assignments with anxiety due to a lack of information about what to expect. In both districts, teachers par- 
ticipating in focus groups reported little confidence in being able to answer questions from students on how they would 
use devices, submit answers, or complete a test session. However, despite the uncertainty staff had prior to test adminis- 
tration, most agreed that they were able to handle the test administration challenges after the first few days of testing. 

■ What are key staffing decisions that Burlington and Revere are considering? During EOY test administration, Burlington 
pulled back the number of district-level IT staff on-site to experiment with how many staff are needed during a given test 
session. In spring 2015, the district is likely to operate with district-level IT staff on-call, but not on-site. School-level tech- 
nology staff— including library and media specialists— will likely provide troubleshooting support. Revere hopes to have 
two to three tech-trained educators working with a school-level IT staff person in each building during future test admin- 
istrations (see additional details in Training section below); district-level IT staff would be available to support multiple 
buildings during test administration. 

See text box on Staffing below fora summary of these district decisions, and questions other districts can consider in light of 
staffing PARCC test sessions. 


STAFFING 

District decisions for 2014-15 

Key questions for other districts 

Both districts are discussing: 

■ Designating fewer district IT staff to PARCC to ensure less drain on IT 
capacity. 

■ Initiating ongoing staffing and training conversations. 

■ Relying on school staff who are supervising test sessions to provide 
basic troubleshooting rather than assigning additional staff to that 
role. 

■ Who is qualified to perform the sophisticated IT tasks needed to 
prepare a district for computer-based test administration? 

■ Who will make decisions on scheduling test sessions? 

■ Who will take lead responsibility for managing devices, including 
needed updates? 

■ Who will staff test sessions as test administrators? 

■ How many staff are needed for each test session? 


Orientation and professional learning for educators 

Prior to PARCC testing, most educators reported that they did not have extensive experience with computer-based testing or 
software. According to survey data, about 40 percent of educators in Burlington and Revere reported giving computer-based 
tests within the last school year. Among those that did, educators reported giving one or two computer-based tests during the 
past school year. 

■ How did districts prepare educators for computer-based testing? District teams primarily focused on orienting teachers 
to PARCC computer-based testing. In Burlington, district IT staff held orientation meetings with school leaders and teach- 
ers. These groups watched the PARCC tutorial video together, and district IT staff answered questions. The Burlington 
Educators’ Association conducted an afternoon training session for teachers to practice with the tools and share feedback 
and an evening session for parents— co-hosted by Burlington Public Schools— that included demonstrations of computer- 
based test materials. In Revere, school principals held their own building-level orientation meetings for teachers and 
instructional staff. Revere principals tried to focus on getting the most critical information about how the test would func- 
tion to their colleagues, creating “cheat sheets” from the voluminous materials put out by Pearson. Similar to Burlington, 
Revere also held a parent information meeting to inform parents about the PARCC field test. 

11 The staffing of test sessions at Burlington High School was different than what is described for the rest of the district. At Burlington High School, perma- 
nent and per diem substitutes supervised test sessions. 
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■ What were educator perspectives on the usefulness of the test administration training and materials? Educators reported 
that neither training materials made publicly-available by PARCC, nor district-based orientation activities, created a sense 
of how to troubleshoot technology issues. More than 60 percent of educators who served as test administrators reported 
that the PARCC online trainings did not provide preparation to resolve basic technology issues (see Figure 5; Appendix A 
offers additional data). As one teacher attending a focus group said: “I feel like an online simulation would have been bet- 
ter that what was offered [from Pearson]. And there were so many tech problems on the first day, that I don’t feel like [the 
training] was a full-scale simulation.” 

Some teachers reported value in taking the student tutorial with colleagues to identify test administration challenges, 
and what possible solutions existed. Both districts engaged in this type of training: at the district-level in Burlington, and 
at each school in Revere. 


Figure 5. Percent of test administrators reporting on effectiveness of PARCC orientation.* 


The test instructions within the Test Administrator Manual, including 
scripts for PBA test administration, were easy for me to implement 

The policies and procedures within the Test Administrator 
Manual were easy for me to understand. 

The PARCC online trainings prepared me to resolve basic problems 

related to technology. 

The PARCC online trainings effectively informed me of the protocol 
to follow during PBA test administration. 



0 10 20 30 40 50 60 70 80 90 100 


■ Burlington 

■ Revere 


*Figure 5 reports on the percent of educators with a test administration role who responded “Strongly agree ” or “Agree” with statements on PARCC 
trainings. 


■ What are key training decisions that Burlington and Revere are considering? Both Burlington and Revere are planning to 
provide more IT training to teachers in order to reduce the number of IT staff assigned to schools during the testing. In 
Revere, district-level IT staff are considering how to train a select number of teachers to be “tech trouble shooters” dur- 
ing future test sessions. In both cases, these teachers would serve as a “first line” of on-site IT support, bolstering support 
provided by school-level IT staff, and freeing up district IT staff assigned to the school during the field test. 


See text box on Training below for a summary of these district decisions, and questions other districts can consider in light of 
offering educators new, or additional, training. 


TRAINING 

District decisions for 2014-15 

Key questions for other districts 

Burlington is planning to differentiate training options for district IT 
staff, building IT staff, and teachers. All training will include a focus on 
which staff members are responsible for what level of troubleshooting. 

Revere will opt to provide a small cohort of teachers basic IT training so 
that they can assist in troubleshooting. 

■ Who is responsible for training educators? 

■ What kinds of training— content and format— will test administrators 
receive? 

■ How and when will the training be provided? 


Rennie Center for education research & policy 


13 




Student experience with PARCC tests 

While both PARCC and MCAS are aligned with the Common Core, the skills required of students to take the computer-based 
PARCC assessment are different from MCAS. The PARCC field test enabled students and teachers in Burlington and Revere to 
make some very preliminary observations about student preparation for, and reaction to, computer-based PARCC tests. In 
opting to conduct district- and school-wide trials, Burlington and Revere sought to strike a balance; school leaders and teach- 
ers wanted students to be informed about PARCC, and take the tests seriously, but not have the trial overtake all other learn- 
ing activities and commitments. Using this approach, school leaders and teachers worked together to set expectations for 
students about computer-based test sessions. 

■ How did educators prepare students for computer-based testing prior to the trial? Educators reported investing class time 
in publicly-available PARCC preparation activities and materials. According to survey data, about three-quarters of educa- 
tors across both districts reported using class time to have students watch the PARCC tutorial, and 70 percent reviewed 
publicly-released test items themselves. Finally, nearly a quarter of educators reported using publicly-released PARCC 
items to design a lesson. 

Burlington and Revere educators communicated with students about the purpose of the PARCC tests. Educators in 
Burlington told students that the tests did not “count,” and this was their opportunity to “test the test” and “preview” 
new expectations for their learning. In Revere, students were expected “to do their best,” but they expressed an innate 
sense that the test did not bear the same weight as MCAS. One student reported: “I knew it was not going to be graded in 
the same way.” 

■ What happened during the trial? Students who participated in focus groups reported on what it was like to take PARCC 
tests on a device, as well as the content of the tests. In regard to using new technologies, student opinions varied largely 
by age. Older students preferred the paper and pencil “MCAS format,” while younger students reported that testing on 
the computer was “fun” and that it was “easier to type” than write, even though many students have not yet taken a key- 
boarding/typing class. 

When considering the content of the PARCC tests, student opinions gathered from focus groups also varied. Students 
reported that practice items— which they took as part of the online tutorial with their class— were not very helpful as the 
content of these did not seem representative of questions on the actual test. Students participating in the focus groups 
reported that the math questions were more challenging than MCAS test items, and reported on the challenges of using 
math test tools, like the equation editor. Some students reported being confused by what the test was asking them to 
do, for example, “[not] knowing how much work to show” when entering math answers, whether work on their scrap 
paper would “count like it had on MCAS,” and “...to write in the box” when referring to the writing prompts on the English 
Language Arts test.” 

■ What were educator perspectives on students' readiness to take PARCC? On the pre-test, more educators at the elemen- 
tary- and middle-school levels reported students having needed skills to take a computer-based test than on the post-test 
survey (see Figure 6, and Appendix A for additional data). Similarly, teachers participating in focus groups reported seeing 
students using “computer skills” as needed to manipulate devices, and witnessed students easily using features like a 
mouse or keyboard keys to answer questions. 

■ What decisions are Burlington and Revere considering to support students' test-taking experiences? I n the focus grou ps, 
Burlington and Revere teachers discussed different strategies to increase the use of technology in their classrooms. 
Importantly, their focus is on building skills and competencies to support student learning, not test preparation. Teachers 
also want to make sure that classroom practice is adjusted to incorporate computer skills that would, in fact, enhance 
student learning. Since these districts have tackled fundamental issues like investments in technology infrastructure and 
staff capacity, these conversations about classroom integration are possible. 
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Figure 6. Percent of educators reporting on students’ technology readiness. 



ELEMENTARY MIDDLE HIGH (Burlington only) 


■ Only a few of the students in my class had the 
needed computer skills. 

■ About half of the students in my class had the 
needed computer skills. 

Most, but not all, of my students had the needed 
computer skills. 

■ All my students had the needed computer skills. 


*Figure 6 reports on the percent of educators that reported at least some knowledge on students’ technology readiness, and does not include “I don ’t 
know” responses. 


See text box on Technology Integration below for a summary of district decisions, and key questions for other districts to con- 
sider when considering issues technology integration in classrooms. 


TECHNOLOGY INTEGRATION 

District decisions for 2014-15 

Key questions for other districts 

Both districts have focused on classroom-level technology integration, 
making this a priority by: 

■ investing in technology resources, 

■ offering teachers professional development, and 

■ using technology more regularly as part of instructional activities. 

■ How routinely are instructional tasks or classroom assessments 
performed by students using technology? 

■ What types of devices are used for these types of activities? 

■ Which educators in your district are addressing issues of technology 
in the classroom? 

■ To what extent is technology integration for classroom instruction a 
critical issue for district leaders? 
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Lessons learned from the 2014 PARCC trials 

District- and school-wide trials in Burlington and Revere produced important information that can help districts weigh the 
pros and cons of computer-based testing. A careful review of lessons learned from both districts’ decision-making processes 
reveals a number of key issues that warrant analysis by all school districts across the Commonwealth considering computer- 
based PARCC tests. 

Determining technology infrastructure capacity and the inventory and location of PARCC-ready devices ahead of 
time is critical. Burlington and Revere have prioritized the use of technology in schools. Over multiple years, they committed 
resources to strengthening infrastructure, improving high-speed connectivity, and hiring experienced IT staff. To achieve these 
goals, both districts have established effective partnerships with their communities, or accessed additional federal or private 
resources. As such, both of these districts had more than adequate technology infrastructure features to support a district- or 
school-wide trial. 

However, Burlington and Revere both expressed concern with maximizing the number of devices that can be used for student 
testing. While PC desktops were considered optimal for testing, neither district is considering this type of expansion in their 
technology inventory. Burlington is likely to use mobile devices to test students in their grade-level classrooms in spring 2015. 
Revere is considering the purchase of Chromebooks to ensure a sufficient number of testing devices to complete test adminis- 
tration within the three week window and minimize over-reliance on computer labs. Perhaps most importantly, both districts 
preferred mobile devices for the ability to support student learning activities outside of testing. 

Burlington and Revere will also focus on improving wireless internet coverage in what will be high-demand sections of build- 
ings; during the trial, both districts tried to cluster test administration in certain wings of school buildings, and will continue to 
do so to the extent possible in future test administrations. This may require the purchase of additional wireless access points. 
For example, Revere noted some classroom locations within testing buildings had lower levels of wireless connectivity during 
Performance-based Assessment test sessions. The district purchased more wireless access points for buildings, which were 
used successfully during End-of-Year tests. 

School-level management of test administration is challenging, Districts want to minimize the number of days that 
are dedicated to testing in each building, and across the district. The total time devoted to testing entirely depends on the 
inventory, type, and location of available devices; unlike MCAS, not all students can be tested at one time. School leaders in 
Burlington and Revere needed to create a school-wide schedule to maintain and carry out “regular” school functions for multi- 
ple days during the testing period. This required planning for instances when space, staff and technology resources were often 
limited or unavailable due to test administration. 

Looking ahead to spring 2015, Burlington is likely to use their full inventory of mobile devices plus laptops and desktops, total- 
ing over 4000 devices, to test students in grades 3-8 in as few days as possible. The district will try to achieve this target by 
testing, in grade-level groups, as many students as possible in concurrently scheduled sessions (e.g., all third graders across 
the district will take the first session of PBA math at a particular date/time). Revere is weighing several scheduling scenarios. 
District leaders are determining different configurations to concurrently test as many students as possible to maximize device 
use across buildings. Revere is also considering the purchase of additional Chromebooks to complete testing sessions more 
quickly, and as a “back-up inventory.” 

Creating a variety of test administration roles for staff will require more support and training. During PBA and EOY test 
administrations, Burlington and Revere had educators and IT staff handling new and different responsibilities than what had 
been the case during MCAS test administrations. District IT staff were heavily involved in planning test administration, and in 
some cases, training colleagues; teachers were involved in troubleshooting some technological issues during test sessions. 
District teams experimented with informal training protocols to simultaneously increase staff knowledge toward new respon- 
sibilities, and to determine how much IT staff capacity is needed once teachers became more comfortable with technology. 
Burlington ultimately pulled back the number of district IT staff on-site during EOY test sessions administered by teachers, 
relying on building-level IT staff as a first line of support for technology challenges. 


16 


In Revere, district-level IT staff are considering how to train a select number of teachers to serve as “tech trouble shooters” 
during test sessions. The district desires to have two to three tech-trained staff per building to ensure that district-level IT staff 
are free to support multiple buildings during test administration. Regardless of approach, teachers in these two districts will 
likely have new technological responsibilities in test administration, and district-level IT staff will have more responsibility in 
planning for test administration as well as training colleagues. Therefore, new supports for teachers and IT staff are a critical 
priority. 

Investments in technology were and will continue to be focused on instruction* Burlington and Revere opted into PARCC 
district- and school-wide trials based on the view of district leadership that technology-enhanced instruction and assessment 
is “the wave of the future.” The PARCC field test provided an opportunity to determine how to manage a computer-based 
test in terms of planning for test sessions, staffing, assessing needed training, preparing students and managing district- and 
school level resources. District leadership has clearly articulated to their respective teams, however, that assessment is only 
one of the educational activities where technology will play an increasingly expanded role; both districts are determined to 
ensure that the primary focus of all decisions about technology is the improvement of student learning. 

Given that most students did not report significant roadblocks utilizing technology during the PARCC field test, many edu- 
cators in these districts are optimistic about the integration of technology into classroom practice. Educators in Burlington 
and Revere were vocal about keeping the focus on student learning as they spoke about the field test and its implications. 
Teachers were positive and thoughtful about the potential of technology for improving classroom instruction and classroom- 
based assessment, so long as the goal of technology integration remains on improving student learning. 
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Questions for Further Investigation 

The Burlington and Revere school districts have an established track record of substantial investments in infrastructure, 
equipment and staff to support classroom-level integration of technology. Despite their technology capacity, however, both 
districts recognized the challenges of implementing computer-based assessments at scale, and opted into school- and district- 
wide field testing to learn as much as possible about how test administration would affect regular school operations. 

During the field test, these districts experienced challenges with PARCC administration including delivery glitches in the first 
days of testing, disruptions to classroom instruction, and the need to assign staff to perform additional and new roles. Both 
district leaders, however, focused on what learning could be gained from overcoming these challenges, for the dual purposes 
of completing test administration and improving technology implementation in the classroom. This case study raises ques- 
tions about the intersection of technology fortesting and technology for teaching and learning, and suggests that districts 
looking to adopt computer-based testing carefully review how technology for testing impacts: 

■ technology infrastructure and capacity; 

■ budgetary resources for upgrading and expanding equipment inventory; 

■ facilities, including classroom space, and their availability during testing windows; 

■ staffing requirements and capacity, in light of new and different assignments for educators and technology staff, alike; 

■ professional development needs; and 

■ student learning in classrooms. 


These important questions about technology needs warrant more study given that the Commonwealth of Massachusetts is 
facing important decisions on the future of computer-based testing. 

Finally, the case study raises questions beyond the implementation of computer-based testing that concern the impact of 
technology on teaching and learning, and resource equity. Both districts viewed the field test as a first step toward a bet- 
ter understanding for them of how to integrate technology more effectively into classroom instruction. There is no dispute 
that differences exist in resources, including infrastructure and devices, among school districts. Further investigation of 
these questions would inform future steps Massachusetts may take to more fully equip schools and classrooms— across the 
Commonwealth— with improved digital capacity and access. Only in reaching greater levels of digital parity will the potential 
of technology to support teaching and learning be possible. 
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APPENDIX A: 


Key findings from the PARCC educator survey 

Reported below are selected findings from the pre-test and post-test surveys administered to educators as part of the case 
study documenting computer-based testing during the PARCC field test in Burlington Public Schools and Revere Public 
Schools. This case study, focusing on the these two districts’ experiences in implementing computer-based testing district- 
and school-wide, was conducted by the Rennie Center for Education Research & Policy. 

Important information to note about survey results presented below: 

■ In Burlington, all teachers and instructional staff were surveyed. 

■ In Revere, all teachers and instructional staff in the three testing schools were surveyed. 

■ Responses are reported for all teachers who responded to the survey. 

■ Tables 5-8 present findings from survey questions where teachers indicated that they had some role in test administration 
(i.e., test administrator, proctor, test coordinator) for the PARCC Performance-based Assessment (PBA) 1 , and therefore 
does not represent results from the full sample of teachers who completed the survey. 


Table Al. Number of survey participants 



PRE-TEST 

POST-TEST 

Item 

Burlington 

Revere 

Total 

Burlington 

Revere 

Total 

In which district do you work? 

293 

90 

383 

179 

78 

257 


Table A2. How many years have you been teaching in this district? 




PRE-TEST 



POST-TEST 


Response 

Burlington 

Revere 

Total 

Burlington 

Revere 

Total 

In which district do you work? 

293 

90 

383 

179 

78 

257 

1-3 years 

16% 

14% 

15% 

13% 

36% 

15% 

4-10 years 

39% 

40% 

39% 

39% 

37% 

38% 

More than 10 years 

38% 

35% 

37% 

41% 

9% 

40% 


Table A3. Do you teach in a content or grade tested by the PARCC performance-based assessment? 

Response 

Burlington 

Revere 

Total 


Yes 

59% 

72% 

63% 

No 

41% 

26% 

36% 


1 The PARCC PBA is a summative test that occurs after 75 percent of the school year to measure student knowledge. The PARCC End-of-Year assessment 
(EOY) occurs after the completion of about 90 percent of the school year. Results from both tests are combined to produce students’ summative assess- 
ment scores. 
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Table A4. Prior to test administration, did you review any PARCC test items/materials? 



PRE-TEST 


POST-TEST 

Response 

Burlington 

Revere 

Total 


Revere 

Total 

Yes 

57% 

79% 

68% 

60% 

90% 

69% 

No 

43% 

21% 

32% 

39% 

9% 

30% 

If yes, How much time did you spend reviewing such materials ? 

30 minutes or less 

32% 

9% 

20% 

4% 

25% 

17% 

More than 30 minutes to about an hour 

28% 

20% 

24% 

17% 

32% 

26% 

More than an hour, but less than 2 hours 

25% 

39% 

32% 

31% 

27% 

29% 

More than 2 hours, but less than 3 hours 

8% 

13% 

10% 

20% 

6% 

n% 

3 hours or more 

7% 

20% 

13% 

26% 

9% 

16% 


Table A5. Among those who had some assigned role in test administration: The PARCC online trainings effectively 
informed me of the protocol to follow during PBA test administration. (Post-test data only) 


Response 

Burlington 

Revere 

Total 

Strongly agree 

6% 

4% 

5% 

Agree 

47% 

59% 

52% 

Disagree 

32% 

33% 

33% 

Strongly disagree 

4% 

3% 

3% 

N/A (1 did not review these resources before PBA administration) 

9% 

1% 

6% 


Note: This item was administered only to those who responded they had a specific role in the PBA test administration in spring 2014. 


Table A6. Among those who had some assigned role in test administration: The PARCC online trainings prepared me to 
resolve basic problems related to technology. (Post-test data only) 


Response 

Burlington 

Revere 

Total 

Strongly agree 

2% 

1% 

2% 

Agree 

24% 

35% 

28% 

Disagree 

42% 

44% 

43% 

Strongly disagree 

20% 

19% 

20% 

N/A (1 did not review these resources before PBA administration) 

12% 

1% 

8% 


Note: This item was administered only to those who responded they had a specific role in the PBA test administration in spring 2014. 


Table A7. Among those who had some assigned role in test administration: The device a student was using did not appear 
to be working during test administration. (Post-test data only) 


Response 

Burlington 

Revere 

Total 

Yes 

76% 

87% 

80% 

No 

19% 

12% 

16% 

No technology-related problems occurred during test administration 

4% 

1% 

3% 


Note: This item was administered only to those who responded they had a specific role in the PBA test administration in spring 2014. 
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Table A8. Among those who had some assigned role in test administration: The device a student was using lost internet 
connectivity during test administration. (Post-test data only) 


Response 

Burlington 

Revere 

Total 

Yes 

59% 

72% 

64% 

No 

37% 

24% 

32% 

No technology-related problems occurred during test administration 

4% 

3% 

3% 


Note: This item was administered only to those who responded they had a specific role in the PBA test administration in spring 2014. 


Table A9. Class time/schedule was impacted by the PARCC PBA. (Post-test data only) 


Response 

Burlington 

Revere 

Total 

No impact 

4% 

1% 

4% 

Minor impact 

18% 

29% 

21% 

Significant impact 

77% 

69% 

75% 


Table A10. Room, equipment, or facilities availability was impacted by the PARCC PBA. (Post-test data, only) 


Response 

Burlington 

Revere 

Total 

No impact 

24% 

6% 

19% 

Minor impact 

25% 

21% 

24% 

Significant impact 

49% 

72% 

56% 


Table All. About how many instructional minutes did you devote to PBA test administration this school year, including 
student preparation and practice, as well as test administration time? 



PRE-TEST 


POST-TEST 

Response 

Burlington 

Revere 


Burlington 

Revere 

Total 

1 hour or less 

31% 

8% 

19% 

25% 

6% 

19% 

More than 1 hour, but less than 2 hours 

47% 

28% 

38% 

11% 

10% 

11% 

More than 2 hours, but less than 3 hours 

17% 

35% 

26% 

3% 

8% 

4% 

More than 3 hours, but less than 4 hours 

3% 

17% 

10% 

3% 

12% 

5% 

4 hours or more 

3% 

12% 

7% 

17% 

41% 

25% 

1 don't teach in a grade or content area in 
which the PARCC PBA was administered 

0% 

0% 

0% 

42% 

23% 

36% 


Table A12. How does this amount of time on PARCC preparation and administration compare to the time devoted to MCAS 
preparation and administration? (Post-test data only) 


Response 

Burlington 

Revere 

Total 

More time working on PARCC preparation this school year than MCAS in previous school years. 

7% 

37% 

16% 

About the same amount of time with PARCC and MCAS items 

13% 

26% 

17% 

More time preparing for MCAS in previous school years than PARCC this school year. 

33% 

15% 

28% 

N/A (e.g., 1 have had different roles/responsibilities in preparing for MCAS and PARCC tests) 

46% 

21% 

39% 
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Table A13. How does the amount of time spent reviewing PARCC test items compare to time spent working with MCAS 
publicly-released items and practice tests? (Pre-test data only) 


Response 

Burlington 

Revere 

Total 

More time working on PARCC preparation this school year than MCAS in previous school years 

23% 

33% 

28% 

About the same amount of time with PARCC and MCAS items 

11% 

26% 

18% 

More time preparing for MCAS in previous school years than PARCC this school year 

28% 

29% 

28% 

N/A (e.g., 1 have had different roles/responsibilities in preparing for MCAS and PARCC tests) 

38% 

13% 

26% 


Table A14. In your opinion, how ready are/were students to take the PARCC PBA administration? 




PRE-TEST 



POST-TEST 


Response 

Burlington 

Revere 

Total 

Burlington 

Revere 

Total 

All my students had the computer skills they needed 
to take a computer-based test 

14% 

8% 

11% 

28% 

27% 

28% 

Most, but not all, of my students had the computer 
skills they needed to take a computer-based test 

34% 

50% 

42% 

27% 

51% 

35% 

About half of the students in my class had the comput- 
er skills they needed to take a computer-based test 

13% 

26% 

19% 

9% 

17% 

11% 

Only a few of the students in my class have the com- 
puter skills they needed to take a computer-based test 

9% 

10% 

9% 

8% 

4% 

7% 

1 don’t know enough about test administration pro- 
tocols 

30% 

7% 

18% 

26% 

1% 

19% 


Table A15. In your opinion, how ready are/were students to take the PARCC PBA administration? (Results reported by 
grade-level taught) 




PRE-TEST 



POST-TEST 


Response 

Elem. 

Middle 

High 

Total 

Elem. 

Middle 

High 

Total 

All my students had the computer skills they needed 
to take a computer-based test 

4% 

16% 

23% 

14% 

18% 

36% 

29% 

28% 

Most, but not all, of my students had the computer 
skills they needed to take a computer-based test 

29% 

45% 

41% 

38% 

37% 

39% 

16% 

31% 

About half of the students in my class had the comput- 
er skills they needed to take a computer-based test 

23% 

13% 

8% 

15% 

14% 

11% 

5% 

10% 

Only a few of the students in my class have the com- 
puter skills they needed to take a computer-based test 

20% 

3% 

1% 

8% 

12% 

3% 

8% 

7% 

1 don’t know enough about test administration pro- 
tocols 

24% 

23% 

27% 

25% 

19% 

11% 

42% 

24% 
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Table A16. 1 believe the technology resources in my school [are ready for/worked well during] PARCC PBA administration. 




PRE-TEST 



POST-TEST 


Response 

Burlington 

Revere 

Total 

Burlington 

Revere 

Total 

Strongly agree 

23% 

6% 

15% 

9% 

6% 

8% 

Agree 

44% 

57% 

51% 

35% 

63% 

44% 

Disagree 

12% 

20% 

16% 

29% 

24% 

28% 

Strongly disagree 

3% 

2% 

3% 

10% 

4% 

8% 

1 don’t know enough about the administration 
protocols 

16% 

15% 

15% 

16% 

1% 

12% 


Table A17. 1 believe the plan for administration and logistics in my school [are ready for/worked well during] PARCC PBA 
administration. 




PRE-TEST 



POST-TEST 


Response 

Burlington 

Revere 

Total 

Burlington 

Revere 

Total 

Strongly agree 

13% 

8% 

11% 

4% 

12% 

6% 

Agree 

41% 

64% 

52% 

35% 

67% 

44% 

Disagree 

20% 

18% 

19% 

31% 

13% 

25% 

Strongly disagree 

10% 

1% 

5% 

17% 

5% 

13% 

1 don’t know enough about the administration 
protocols 

17% 

9% 

13% 

14% 

4% 

11% 
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